Repository navigation
Conversation
… failed hops and classifier bodies in debug log 08:50 incident: one auto-mode classifier request got a bare 429 on 8 accounts in a row; each 429 cooled its whole account for 60s and the pool went eligible=0. - failover.max_accounts_per_request (0 = unlimited): hard stop after N accounts ran - bare 429 on bare_429_request_scope_after (2) accounts of one request -> request-scope stop, nobody cooled - a bare 429 cools the account only after bare_429_account_sessions (2) distinct sessions hit it within bare_429_window_ms; otherwise it only steps aside for the request - failed hops (status, rate-limit headers, body, provider facts) go into the debug record as hop_errors - classifier requests keep inbound/outbound bodies in debug mode
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd260624e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| body: clipText(result?.body), | ||
| provider_result: clipText(result?.providerResult, 2000), |
There was a problem hiding this comment.
Redact credential-shaped strings before storing hop errors
When an upstream or CLI error contains a bearer token or sk-ant-... credential, clipText only truncates it, and RequestLogStore.finish persists the resulting hop_errors verbatim. Unlike the existing clampBody and debugResponse paths, neither this helper nor the new persistence path applies redactSecrets. Consequently, even caller-forced debug mode can leave usable credentials in SQLite and the panel's debug records. Apply the existing secret masking to the hop body and provider facts before retaining them.
Useful? React with 👍 / 👎.
| if (this.accountCapReached(ranAccounts)) { | ||
| releaseStickySlots() | ||
| return stopAtAccountCap(result, policy, ranAccounts.size, budget.hops, attribution()) |
There was a problem hiding this comment.
Enforce the account cap on retry continuations too
With max_accounts_per_request: 1 and two available accounts, an uncommitted empty/incomplete hop takes the preceding allowSameUnit continuation before reaching this guard. That continuation sets avoid to the first account, so the next selection deliberately prefers another account and executes the request on two distinct accounts despite the cap of one. The exception path has the same ordering. Enforce the cap when selecting the next account, including retry continuations, rather than only in the ordinary switch path.
Useful? React with 👍 / 👎.
| hits.push({ key: String(key || ''), at: now }) | ||
| this.bare429Hits.set(accountId, hits) |
There was a problem hiding this comment.
Discard cooldown evidence from request-scoped 429s
With the defaults, let one session receive bare 429s from accounts A and B, then let a second session receive the same responses within 120 seconds. Both requests end as rate_limited_request, but the first request leaves a hit for A in this map. The second request's first hop therefore triggers applyCooldown(A) before B produces the request-scope verdict, and neither the hit nor the cooldown is rolled back. Repeated request-scoped failures consequently still remove accounts from the pool despite the intended "cool nobody" behavior. Defer or discard account cooldown evidence for requests that receive a request-scope verdict.
Useful? React with 👍 / 👎.
|
感谢贡献。替代修复 #1165 已合入 main(b1842a588e)。实际执行账号可设置硬上限;重复裸 429 只结束本次请求,保留有证据的冷却与额度探测。不保存原始 hop / 分类器正文,不使用跨请求猜测。 此 PR 不再重复合并。 |
Problem
One auto-mode classifier request (sonnet, ~140 KB system prompt) got a bare 429 (no
anthropic-ratelimit-*headers, no usable body) on 8 accounts in a row:rate_limited_unknownand cooled its whole account for 60 s, so one request took 8 accounts out of the pool. The pool wentpool_rate_limited eligible=0for about a minute and other clients got 18 × 429.max_account_switches/max_total_attemptswere already set to 3, but they are not hard limits: once they are passed, untried units stay reachable, so the request kept walking accounts.Fix
failover-runner.mjs(all new knobs live underfailoverinrouting.json):max_accounts_per_request(default0= unlimited, unchanged behavior). A hard stop after N accounts ran; the request answers with the last hop, so a bare 429 stays a 429.bare_429_request_scope_after(default2). A bare 429 on that many accounts of one request means the request itself is the trigger: stop withrate_limited_requestand cool nobody.bare_429_account_sessions(default2) withinbare_429_window_ms(default120000). A single bare 429 only steps the account aside for that request. The account cools only once that many distinct sessions hit a bare 429 on it inside the window. 429s with rate-limit headers or a recognised body are handled exactly as before.hop_errors.request-log.mjs: classifier requests now keep inbound/outbound bodies in debug mode.Behavior change
With the defaults, one bare 429 no longer cools an account for 60 s by itself. Set
bare_429_account_sessions: 1to keep the old behavior.Tests
failover-runner.test.mjs: new cases for request-scope stop on two accounts, cooling only on a second session, themax_accounts_per_requesthard stop, and kept hop errors.request-log.test.mjs: classifier bodies are kept, andhop_errorsis in the debug record.Verified in production
Running since 2026-11-16 on 1.4.112 with these files mounted and
max_accounts_per_request: 3.